Skip to content

docs: draft MySQL snapshot parallelism page - #38091

Open
peterdukelarsen wants to merge 3 commits into
mainfrom
pl/mysql-partition-user-docs
Open

docs: draft MySQL snapshot parallelism page#38091
peterdukelarsen wants to merge 3 commits into
mainfrom
pl/mysql-partition-user-docs

Conversation

@peterdukelarsen

@peterdukelarsen peterdukelarsen commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Part of SS-358.

Adding docs for the new MySQL parallel snapshotting feature, some more context on the load we place on upstream databases, and some high-level debugging tips for triaging obvious database issues.

The table-splitting feature ships with mysql_source_snapshot_parallelism off by default, so it is documented as private preview: a banner on the MySQL snapshot parallelism page and inline preview markers where the splitting is mentioned elsewhere.

🤖 Generated with Claude Code

https://claude.ai/code/session_016AgSuGEvoVT9TFHAgq6FCi

@peterdukelarsen peterdukelarsen changed the title pl/mysql partition user docs docs: draft MySQL snapshot parallelism page Aug 6, 2026
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch 2 times, most recently from 4d26654 to c530367 Compare August 6, 2026 23:31
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch 2 times, most recently from 7cd15a2 to 6fb412a Compare August 7, 2026 00:05
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 6fb412a to ad4508e Compare August 7, 2026 14:03
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from ad4508e to bedaa57 Compare August 7, 2026 22:27
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from bedaa57 to 07af68f Compare August 7, 2026 23:04
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch 2 times, most recently from a0bd0d7 to 1d5f164 Compare August 8, 2026 01:44
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 1d5f164 to 4aa343a Compare August 9, 2026 17:35
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 4aa343a to a8438ee Compare August 10, 2026 20:32
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from a8438ee to 529b0be Compare August 10, 2026 20:46
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 529b0be to bdce9de Compare August 10, 2026 21:10
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from bdce9de to b7a0bc2 Compare August 11, 2026 01:40
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from d189c40 to 132f2b4 Compare August 11, 2026 18:58
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 132f2b4 to 1bdd206 Compare August 11, 2026 19:06
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 1bdd206 to 9329cfa Compare August 11, 2026 19:25
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 9329cfa to 1d8fe51 Compare August 11, 2026 19:32
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 1d8fe51 to 891ad5f Compare August 11, 2026 19:32
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 891ad5f to 2167caa Compare August 11, 2026 19:33
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 2167caa to 1805742 Compare August 11, 2026 19:40
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 1805742 to 9efe79a Compare August 11, 2026 20:15
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 9efe79a to b105665 Compare August 11, 2026 20:46
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from b105665 to 7d24859 Compare August 11, 2026 22:41
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from 7d24859 to bc6f2ff Compare August 11, 2026 23:02
@peterdukelarsen
peterdukelarsen force-pushed the pl/mysql-partition-user-docs branch from bc6f2ff to 28fd467 Compare August 11, 2026 23:36
User-facing draft for the parallel snapshot feature: how key ranges
divide across workers, single-column string PK requirement and
fallback, upstream connection and statistics considerations, and where
to observe progress. Marked private preview.
@linear-code

linear-code Bot commented Aug 24, 2026

Copy link
Copy Markdown

SS-358

…ance

Make the MySQL page strictly MySQL-specific (split eligibility,
prefix-based partitioning, connection counts) and move the cross-source
worker-scaling discussion into a shared snapshotting parallelism include
on the snapshotting concept page and the ingest overview. Add a
troubleshooting section on telling whether the upstream database is the
bottleneck (CPU, IOPS, throughput, connections, cache, replica lag) and
what to do about it. Present the feature as production, without a
preview banner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018yzDA9ywnCLXweNCzqBm12
Comment thread doc/user/content/ingest-data/mysql/snapshot-parallelism.md
headless: true
---

Materialize parallelizes snapshotting across the workers of the cluster

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we don't parallelize everything (sql server), I would make it more "can" and separate out. I think something like (?):

Materialize can parallelize snapshotting across the workers of the cluster
hosting the source. 

- **PostgreSQL sources** are parallelized by table, i.e., different tables are
  read concurrently by different workers. On PostgreSQL 14 and later,
  Materialize additionally attempts to partition each table's read across
  workers; tables that cannot be partitioned fall back to a single worker.

- **MySQL sources** are parallelized by table, i.e., different tables are read
  concurrently by different workers. For tables that meet [certain
  requirements](/ingest-data/mysql/snapshot-parallelism/), Materialize
  additionally partitions the table's read across workers. See [MySQL snapshot
  parallelism](/ingest-data/mysql/snapshot-parallelism/).

- **Kafka sources** are parallelized by topic partition, with partitions distributed
across workers.  Parallelism is bounded by the topic's partition count. 

- **SQL Server sources** are not parallelized; a single worker reads all tables.

I might have misinterpreted something ... but, hopefully, the above can give a gist of what I'm thinking

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted your structure, thanks. Only tweak: split the PG bullet's fallback clause into its own sentence.

so parallelism is bounded by the topic's partition count. SQL Server sources
are not parallelized: a single worker reads all tables.

A cluster's [size](/sql/create-cluster/#available-sizes) determines its

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

The degree of snapshot parallelism depends on the number of workers. A cluster's
[size](/sql/create-cluster/#available-sizes) determines its number of workers,
so a larger cluster shortens the snapshot. The volume read from the upstream
database is unchanged; it is compressed into a shorter window of more concurrent
queries and connections. To determine whether snapshotting is overloading the
upstream database, and for ways to mitigate the load, see [Is the upstream
database
overloaded?](/ingest-data/troubleshooting/#is-the-upstream-database-overloaded).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted.


- **Read load.** Snapshotting puts read, CPU, and network load on the upstream
system, proportional to the data volume.
system, proportional to the data volume and concentrated in proportion to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

? Is it more?

- **Read load.** Snapshotting puts read, CPU, and network load on the upstream
  system. The total load is proportional to the volume of data being
  snapshotted, while the [source cluster's parallelism](#parallelism) affects
  the peak load: more workers compress the reads into a shorter window.

?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted the total-vs-peak framing.

- **Network throughput** at the instance type's cap.
- **Memory** pressure, or a falling cache hit rate as large scans evict the
normal workload's working set.
- **Connections** near the database's limit. Snapshotting opens connections

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The snapshotting sentence isn't true/applicable for kafka and sql server, yes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Scoped the bullet to PostgreSQL and MySQL sources.

Comment thread doc/user/content/ingest-data/mysql/snapshot-parallelism.md
## Which tables are split

The snapshot of an individual table is split across workers when all of the
following hold:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Materialize splits the snapshot of an individual table across workers when all of the following conditions are met:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted.

values. See [How a table is partitioned](#how-a-table-is-partitioned).

Tables that don't meet these requirements, or whose boundary sampling fails
for any reason, still snapshot correctly: each is read in full by a single

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we simplify and add a link to the boundary sampling?

If a table does not meet these requirements, or if the boundary sampling fails,
its snapshot is not split: a single worker reads the table in full.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted, with one extra sentence noting that different tables are still read concurrently.


## How a table is partitioned

Materialize partitions a table by the unique leading characters of its

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an eligible table and maybe separate with a little tweak?

Materialize partitions an eligible table using the
leading characters of its primary key values. Before reading the table,
Materialize probes the primary key index to discover key prefixes and uses the
MySQL optimizer's row estimates to gauge how many rows fall under each prefix.
It extends the prefixes as needed to find boundaries that divide the table into
roughly even ranges. The probes are inexpensive point lookups, capped in
proportion to the table's estimated size, so the sampling phase stays negligible
next to the snapshot itself.

Each worker then reads only its assigned range, within the same
consistent snapshot of the upstream database, so the result is identical to a
single-worker snapshot, only faster.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted your two-paragraph version.

Comment on lines +78 to +92

- **Connection count.** While the snapshot is being set up, Materialize
briefly holds up to two connections per worker, plus one. Once reading is
underway, this settles to one connection per worker reading a range, plus
one coordination connection. After the snapshot completes, the source drops
back to a single replication connection. If your MySQL server or connection
pooler enforces a low
[`max_connections`](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_connections)
limit, account for this burst when sizing it.

- **Statistics freshness.** Range boundaries are placed using the MySQL
optimizer's row estimates. Stale statistics don't affect correctness, but
can skew how evenly work divides across workers. Running
[`ANALYZE TABLE`](https://dev.mysql.com/doc/refman/8.0/en/analyze-table.html)
on very large tables before creating the source can improve balance.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep these here for now: both bullets only apply when parallel snapshotting is active, which is private preview and flag-off, so they'd be noise in the general MySQL considerations. Worth revisiting when the feature is on by default.

Mark the table-splitting feature as private preview since it ships
flag-off, restructure the shared parallelism include as a per-source
list and caveat that a larger cluster can shorten the snapshot only as
far as the work parallelizes, scope the troubleshooting connection
guidance per source type, rework the overload mitigations around
upsizing the database or dropping the source and retrying, point
observability at the existing monitoring page, and add discovery blurbs
to the MySQL and PostgreSQL ingest overview pages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016AgSuGEvoVT9TFHAgq6FCi

@kay-kim kay-kim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants